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© Apparatus for processing and reproducing Image Information. 



® Layout analysis is used in operating a digital 
reproduction apparatus. Digital image dot data cor- 
responding to a document image to be reproduced 
are analysed by the apparatus and the layout of the 
document is determined therefrom. This is shown on 
a display. An operator can now select a specific 
layout element, such as a text column, by indicating 
it, and instruct the image processing unit of the 
apparatus to process only that element. The pro- 
cessing operations include colour printing, gradation 
changing, moving and enlargement/reduction. 

A layout analysis method suitable for this ap- 
plication comprises two steps. In a first step, the 
image information is. searched for. clusters of adjoin- 
ing informationbearing image dots. These clusters, 
known as objects, each form a character, graphic 
element, photograph, etc. In the second step, the set 
of clusters is divided in an iterative process into text 
blocks, lines and words. Only objects which repre- 
sent a character are used in these conditions. 
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The invention relates to apparatus for repro- 
ducing an image . comprising operating means, 
means for supplying digital image dot information 
representing said image, a memory for storing im- 
age dot information, means for processing the sup- 
plied image dot information and means for printing 
the processed image dot information on an image 
support. The term "image dot information" in this 
context denotes the density values of pixels (image 
dots) disposed in an orthogonal raster and together 
corresponding to the image. 

Apparatus of this kind are generally known. In 
these known apparatus, a document is scanned by 
means of an optoelectrical scanner and digital im- 
age dot data are generated which correspond to 
the document image. These image dot data are 
- stored in a memory, processed and printed. Digital 
image dot data can also be supplied from another 
device, stored in the memory, and be printed after 
processing. The processing is generally aimed at 
so adapting the image data that a good print is 
obtained, e.g. converting grey value data into hal- 
ftone data, i.e. image data which can only have the 
value black or white for each image dot. 

In . addition. ■ processing may be aimed at 
changing the appearance of the printed image of 
the document according to an operator's choice. 
This is usually referred to as editing. In that case, 
selection means, such as a digitising panel and an 
indicator pen, can be used to indicate an area of 
the document, whereafter a processing function 
instruction can be given to change the appearance 
of the. indicated area. A disadvantage of the con- 
ventional devices is that the areas to be processed 
must be indicated very accurately. If an area to' be 
processed is indicated . carelessly, a part which 
should really also have been processed may easily 
be omitted. 

The object of the invention is to provide a 
copying or printing apparatus suitable for simple 
and rapid processing of a document image, the 
result being a print with an adapted appearance. 

. To this end, the apparatus is characterised by 
segmentation means, connected to the memory, 
for segmenting the digital image dot information 
into layout- elements, the operating means being 
connected to the segmentation means in order to 
receive therefrom results of the segmentation and 
being provided with: 

- means for displaying said results to an oper- 
ator. 

- first selection means for selecting one or 
more layout elements found during the seg- 
mentation. 

- second selection means for selecting a pro- 
^ cessing operation for changing the appear- 
ance of said image, 

and also being connected to the processing means 



for controlling the same in order to process a 
selected layout element in accordance with a se- 
lected processing operation. 

The segmentation means analyse the layout of 
5 the document, i.e., they search in the document for 
the separate characters, words, lines, text blocks, 
and also photographs, graphics, etc., and deter- 
mine the mutual topographic relationship between 
them It is not necessary for the segmentation • 
10 means also to determine the content of the text 
elements and the mutual logical relationship be- 
tween the layout elements. 

The determined layout of the document is 
shown to the operator, e.g. on a display, where- 
75 upon he can select layout elements just as is done 
on the display of a word processor station. In these 
conditions the knowledge acquired of the layout 
helps for selection to proceed properly. For exam- 
ple, if the operator indicates a word, the first selec- 
20 tion means will also select precisely that word. The 
operator can then specify a given processing op- 
eration which will then be carried out on the se- 
lected layout element by the processing means. 
Such processing operations are. for example, ad- • 
25 aptation of the grey value scale or the colour with 
which the selected layout element is reproduced 
on the print. Especially for scanned photographs, 
the introduction of a raster is desirable for high- 
quality reproduction. When this operation is se- 
30 iected for a photo area, the original grey level 
image is converted into a dot image (as in news- 
. paper photographs) for that area. Other processing 
operations come more under the heading of word 
processing, such as changing the orientation (rota- 
35 tion) and even moving a layout element. In this way 
the utility of the editing function is greatly in- 
creased. 

Layout, analysis is well-known in the art. as will 
be discussed hereinafter. It is used in combination 
. 40 with optical character recognition (OCR)-in systems 
for automatic text analysis. Such systems first seg- 
ment a document into blocks of text, such as 
columns. Within such bjocks. lines and separate 
words are then identified, followed by an OCR 

45 . algorithm recognizing the characters. 

When several text blocks appear on a page, the 
logical order of the blocks, i.e. the order in which 
the blocks have to be read, must be determined in 
advance. In complex text documents, such as 

so newspaper pages, the text blocks of one article 
often do not have a simple, position-related order, 
and it may be impossible for the automatic layout 
analysis to find the reading order by itself. In a 
system, disclosed in EP 0 415 373 A, an automati- 

55 cally determined reading order is indicated in a 
displayed image of text blocks recognised by a 
layout analysis section. An operator may then se- 
lect one or more blocks with a cursor and change 
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the predetermined reading order. This selection is 
different from the selection procedure of the 
present invention in that it is aimed at identifying 
text blocks as entities in the logical structure of a 
text document, and is not related to the actual area 
or perimeter of a specific layout element. 

Layout analysis of the kind taking place in the 
segmentation means can be performed in various 
ways. Two methods are described in William Lam 
et al: "Block Segmentation of Document Images 
Using the X-Y Tree Approach", TR 88-14, Dept of 
Computer Science, SUNY Buffalo, NY, June 1988 
and in Stephen W. Lam: "Reading Newspaper 
Text", Proc. of 10th International Conference on 
Pattern Recognition, New Jersey, USA, June 16-21, 
1990, pp. 703-705. In addition, the inventors of the 
apparatus according to the invention have devel- 
oped a third different method, which is described 
below. 

In the document by William Lam et a!., layout 
analysis of a document page is discussed by refer- 
ence to two segmentation algorithms: the top-down 
algorithm and the bottom-up algorithm. 

In the top-down. algorithm, vertical and horizon- 
tal projection profiles are made by projecting all the 
image dots on to the x-axis or y-axis respectively 
and counting their numbers as a function of the 
location on the associated axis. These profiles are 
used to check whether a document page can be 
divided into a set of layout structure elements by 
plane cuts. Plane cuts are permissible only if no or 
substantially no image dots are situated on a cut- 
ting line or band. , 

If each subblock of the document page formed 
by the cuts in one direction is again cut in the 
other direction in the next operation, increasingly 
smaller subbiocks are formed until no more cuts 
are possible. In the case of a text page this means 
that the level of the individual characters has been 
reached. 

The disadvantage of this algorithm is that for 
each new cut required all the image dots must be 
processed to determine a new projection profile, 
and this requires considerable computing time. 

On the other hand, in the case of the bottom- 
up algorithm, described in the publication by Ste- 
phen W. Lam, adjoining image dots are first 
grouped whereupon the group structures thus 
formed are in turn combined to form larger struc- 
tures when they are closer together than a pre- 
determined distance. This usually only takes place 
on the condition that the structures involved are of 
the same characteristic type of information, e.g. 
text information. 

A disadvantage of the bottom-up algorithm is 
that it can recognise the large structures only with 
difficulty because of its detail approach. For exam- 
ple, narrow boundaries which always recur at the 



same place in consecutive lines, are easy to recog- 
nise viewed as a whole but may be completely 
overlooked from the detail perspective. 

Another disadvantage of such an algorithm is 
s that a label once assigned to a group structure with 
respect to the characteristic type of information can 
no longer be changed during the segmentation 
process. For example, an advertising text in a 
photograph will be permanently regarded as text 
10 although practice shows that it is better to treat 
such a text as part of a photograph. Image pro- 
cessing operations such as enlargement or reduc- 
tion, grey value adjustment, compression, etc., 
should, of course be carried out for the entire 
75 photograph, including the text part thereof. 

In the segmentation method used in the inven- 
tion, however, the digital image dot information is 
first searched for objects formed by clusters of 
adjoining informationbearing image dots, and then 
20 further operations are carried out on the set of 
objects in order to divide the page image into 
blocks. This gives a considerable reduction of the 
number of data to be processed, so that the com- 
plete segmentation processing of a document page 
25 takes place much more rapidly than the methods 
known from the prior art. 

In one embodiment, data relating to location, 
measurements and type of information (e.g. char- 
acter, photograph, graphics) are determined for 
30 each object. The layout analysis, which is only 
logical for text areas in a document, can now be 
automatically restricted thereto by using for the 
segmenting operations only objects which repre- 
sent a character. < 
. 35 On the other hand, some editing operations 

should preferably be performed on all layout ele- 
ments of a specific kind. For instance, a rasterizing 
operation as described before should be applied on 
all photographs in a scanned original document. 
40 For that case, the first selection means can also be 
used in a generic fashion, that is, in the instance 
given, all photographs can be selected at once by 
a suitable generic command. — 
A document which is skew with respect to the 
45 directions used in the segmenting operations (i.e. 
the vertical and horizontal directions) is less sat- 
isfactorily segmented . because layout elements 
may overlap one another. It is therefore advisable 
first to examine the set of objects for skewing, prior 
so to the segmenting (cutting) operations, and if such 
skewing is present this should first be eliminated 
by moving the objects. 

The invention will now be explained with refer- 
ence to the accompanying drawings wherein: 
55 Fig. 1 shows a reproduction apparatus for digital 
image information according to the invention, 
Fig. 2 is a block diagram of the functional con- 
struction of the apparatus in Fig. 1, 
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Fig. 3 is a flow diagram of a method of seg- 
menting a document page, 

Fig. 4 is a diagram showing the classification 

procedure for object boxes. 

Fig. 5 shows part of a page to be segmented. 

Fig. 6 shows part of a skew text, 

Fig. 7 shows a control panel of the reproduction 

apparatus shown in Fig. 1 . 
Fig. 1 shows a digital copying machine/printer 
100 provided with a scanner stage 101 for electron- 
ically scanning a document, a display unit 102, an 
indicator element or light pen 103 for indicating 
locations on the display, a control panel consisting 
of two parts 104 and 105, and a tray 106 to receive 
prints, and finally a processing stage 107. The 
machine can be connected by a cable (not shown) 
to one or more external digital image dot informa- 
tion sources. Of course another indicator element, 
such as a mouse, coupled to a cursor on the 
display may be used instead of the light pen 103. 
A touchsensitive display may also be used so that 
an operator can indicate the various elements with 
his finger. 

Digital grey value image information is ob- 
tained by scanning a document in the scanner 
stage 101 and is stored in an internal memory, if 
required, image information can also be received 
from an external source, e.g. a computer or work 
station, and stored in the internal memory. The 
digital image information is subjected to halftoning, 
so that it is put into the form of a raster pattern of 
image dots which are white or black, and can then 
be printed in the processing stage 107 on a sheet 
of*paP er which is then deposited in tray 106. The 
construction and operation of a processing stage of 
this kind is public knowledge and does not form 
part of the invention and will not therefore be 
explained further. 

The control panel consists of a part 104 for the 
normal copying and printing process and a part 
105 for operating the layout analysis and image 
processing coupled thereto. This function will be 
described in detail with reference to the control 
means on panel 105 in Fig. 7. The part 105 of the 
control panel can of course also be in the form of 
an image made visible on the display, while the 
functions indicated on the panel can be selected 
by indicating them with the indicator element. 

Fig. 2 illustrates the functional construction of 
the apparatus 100 in the form of a block diagram. 
Supply unit, 201, e.g. corresponding to scanner 
101. is connected to image memory 208. Also 
connected to memory 208 are the segmentation 
module 209, processing module 212 and printing 
unit 207, and these three elements are also con- 
nected to operating unit 211. The latter has a 
display 202 corresponding to display unit 102, a 
first selection unit 203 corresponding to indicator 



unit 103, a second selection unit 205 correspond- 
ing to control panel 105 and an apparatus operat- 
ing unit 204 corresponding to control panel 104. 
Segmentation module 209 is also connected to a 
s memory 210. 

The operation of the segmentation module 209 
will first be discussed, followed by the function and 
operation of the other elements of the apparatus. 
Fig. 3 is a flow diagram to illustrate a method • 
jo of segmenting a document page. 

In this flow diagram, step 2 represents the 
dotwise scanning of a document page. 

The information obtained by scanning is digitis- 
ed in step 4 to a multibit torm of representation. 
T5 Steps 2 and 4 describe the generation of the 

image data and thus do not form part of the actual 
segmentation process. They have been added to 
give a complete picture of the total path covered 
by the image data through the system. 
20 The brightness information concerning the 

page background is then removed from this image 
information in step 6. so that only the information- 
bearing image dots or pixels remain. 

This can be effected in a number of ways. One 
25 possible way is to make a grey value histogram of 
the information of the entire page (or part thereof). 
In this histogram, a narrow tall peak situated on the 
light side of the histogram will represent the bright- 
ness of the background and by reference to this a 
so threshold value is determined with which the image 
information is then compared dotwise. image dots 
whose brightness is situated beneath the threshold 
value are removed, so that the background is elimi- 
nated. 

55 In step 8 the image information is converted to 

binary form and stored in a memory. file (bit map). 
This conversion is done by reference to a threshold 
value, for which the threshold value obtained in 
step 6 can be used. In that case the steps 6 and 8 

40 can be combined. - . 

At this moment the number of 'data to be 
processed can be reduced. Many scanners operate 
with a resolution of about 16 dots per millimetre 
(400 dpi) but this resolution often appears unnec- 

45 essary for a correct segmentation. In practice, even 
a resolution of 2 dots per millimetre appears to 
give good results often enough. A reduced resolu- 
tion of this kind can be obtained, for example, by 
making a regular selection from ail the image dots, 

50 e.g. by using only a single image dot from each 
block of 8 x 8 image dots. i.e. storing it in the bit 
map. The further procedure described hereinafter 
is then performed on the reduced image dot set. It 
will be clear that this gives a considerable gain in 

55 processing speed. 

Step 10 then searches for clusters of con- 
nected pixels, i.e. image dots which adjoin one 
another or which are connected through the agency 
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of one or more adjoining image dots. A cluster of 
this kind is hereinafter referred to as "object". 

This can be carried out. for example, as an 
iterative process, each search covering image dots 
adjoining an image dot in a 3 x 3 image dot 
environment. 

At the same time, hence in the same step, a 
rectangular frame or "object box" is defined for 
each object of connected image dots to enclose 
the object. The sides of this frame are parallel to 
the horizontal and vertical direction (the X and Y 
directions respectively in the document page). The 
position of each box thus found is stored in a 
memory file (the object file) in the form of the 
coordinates of the top left and bottom right corner 
points, the number of informationbearing image 
dots situated therein is also stored for each box. 

In the next step 12, each box is assigned a 
label, its dimensions and the number of image dots 
situated therein determining whether the object in 
the box relates to a character, photograph, graph- 
ics, line or noise. 

This classification is based on a number of 
empirical rules which appear to be adequate in 
practice. All the possibilities - are traversed in a 
graduated test procedure shown in Fig. 4. The 
consecutive tests are indicated by an encircled T. 
Two arrows extend from each test, the arrow to the 
left being applicable if the test gives an affirmative 
answer and the arrow to the right in the event of a 
negative answer. 

The tests make use of a reference measure- 
ment defined as the height of a font (character set) 
somewhat larger than the font normally used for, 
texts, namely 12point. If a text with an abnormally 
large font is analysed, then the reference measure-, 
ment must be adjusted, but in practice the same 
font size is used* for a considerable amount of 
printed matter so that the reference measurement, 
selected is in most cases adequate. 

Test T1 checks whether the ratio of the longest 
and shortest side of the object box under examina- 
tion is large and also whether the shortest side is 
smaller than half the reference measurement. If so, 
the object box is given the label "line" (I). 

Test T2 determines the degree of coverage of 
the object box, i.e. the number of information- 
bearing image dots divided by the total number of 
image dots in the object box. If this is very low, the 
object box is classified as "graphics" (g). Object 
boxes clearly larger than the reference measure- 
ment and having a relatively low degree of cov- 
erage are also classified as "graphics". 

In test T3 the remaining . object boxes are di- 
vided into small object boxes (half as large as the 
reference measurement) and large object boxes. 

In test T4 the small object boxes are divided 
into "noise" (n) (if they are very small) and "char- 



acter" (k). 

The large object boxes of test T3 are checked 
for their measurements in test T5. Very large ob- 
ject boxes (more than three times as large as the 
s reference, measurement), are assigned the label 

"photograph" (f). . . o 

In test T6 the remaining large object boxes 
which are smaller than the reference measurement 
in one direction and somewhat larger in the other 
70 direction are provided with the label "character" 
(k). These are presumably two characters stuck 
together. 

The last remaining object boxes are divided in 
test T7 into (small) photographs (f) if their, degree 
76 of coverage is very high, ancj (large) characters (k). 

Some corrections are performed on the clas- 
sification results in the following steps 14, 16 and 
18. 

Step 14: A character object whose box over- 
20 laps a box of a photoobject is reclassified as a 
photoobject. This prevents texts in a photograph 
from undergoing a different image processing op- 
eration from the photoobjects. 

Step 16: When a document page contains one 
25 or more bar codes, it is possible, during the clas- 
sification, that some of the lines are classified as 
line objects but that the others are classified as 
character objects. If. in a set of a minimum number 
of consecutive objects, positioned horizontally or 
30 vertically, of the character type, on the one hand, 
and of the line type, on the other hand, at least a 
predetermined number of line objects occurs, then 
a reclassification of objects takes place such that 
each object in this set is classified as a line object. 
35 Step 18: Very small object boxes which, how- 

ever, are just too large to be eliminated as noise in 
test T4, are divided up into small objects with 
significant information (such as dots and diaereses, 
etc., above letters and dots as part of a punctuation 
40 mark) and small objects without meaning, such as 
noise dots. The test criterion here is whether a * 
character box is found within a predetermined 
small distance from such a small object. If so, this 
small object and the character box thus found are 
45 combined to form a new character box. 

Step 20 then checks whether the image in- 
formation originates from a skew original. This term 
is used to indicate that the document page to 
which the image. data relate is rotated, through a 
so certain angle. This angle is termed the skew angle. 

An accurate method of determining the skew 
angle based on the recognition of the lefthand 
margin of the text on the document page will be 
explained hereinafter with reference to Fig. 6. 
55 In the event of a skew angle, the next step 22 

reorients the set of object boxes. Reorientation of 
the boxes is effected by rotating about a fixed point 
the top left corner point of each object box through 
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an angle which is the opposite of the skew angle. 
The other corner points of such an object box 
undergo identical displacement so that the orienta- 
tion of an object box does not change during this 
translation. . 

Step 22 completes the phase of the method for 
constructing a set of correctly oriented object 
boxes with an associated object label from a bit 
map. 

The objects classified as character are select- 
ed in step 24, step 26 then determining text blocks, 
lines and words in producing a layout structure of 
the text part. The selection of just the character 
boxes means that other object boxes are simply 
cancelled out in this determination and cannot thus 
act as a disturbing factor to frustrate the determina- 
tion of the layout structure. 

In step 26 the document page is cut, so that it 
is divided into rectangles by applying cuts alter- 
nately in the X direction or the Y direction. Starting 
with the smallest rectangle in which all the char- 
acter objects fall (the page block), each block is 
split into smaller blocks at places where cuts can 
be made in the K or Y direction. A cut in the X 
direction in a block to be cutted is placed where a 
horizontal area occurs with a defined width within 
which there is no single character object. 

Similarly, a cut in the Y direction is made in a 
block where there is a vertical area of a certain 
width in which there is no single character object. 
The cuts in the X and Y directions are performed 
alternately. Thus a block formed as a result of a 
horizontal Cut is then cut vertically. If no cuts can 
be made in the X arid Y directions in a block, an 
end block is found. By varying the width that a cut 
must satisfy an end block of this kind can repre- 
sent a paragraph, line of text, word or character. It 
should be noted that when starting to search for 
, possible places for making cuts a search must first 
be made in a given, e.g. horizontal, direction and 
then in the vertical direction, even if no cut was 
possible in the horizontal direction, such as may 
occur, for example, in the case of text arranged in 
columns. 

Determination of, for example, a vertical cut in 
a block B will now be explained with reference to 
Fig. 5. 

The document page P on which the block B 
occurs is formed by character object boxes b(i), 
with i running from 0 (the top left corner) up to and 
including n (the bottom righthand corner of the 
page). The coordinates of the top left and bottom 
* right corners of each box are stored in an object 
file in the memory. 

The block B is formed by the object boxes b(j), 
with j running from p to q and p and q both being 
situated between 0 and ri. 



The object boxes of the block B are first sorted 
by ascending X-coordinate from their top left cor- 
ner and for a constant X-coordinate by ascending 
Y-coordinate from their top left corner. Standard 
5 algorithms are available for this. The resulting se- 
ries in which the object boxes are further indicated 
by b'(i), thus starts with the box situated furthest to 
the left and, if different boxes have exactly the 
same X-coordinate, the topmost of those boxes. 
to For each following object box from the series, 

the smallest enveloping rectangle O of the series is 
then determined, i.e. the rectangle with sides par- 
allel to the x and y axis which just contains al| the 
object boxes of the series, and then the horizontal 
75 distance w of the lefthand side of said object box 
and the enveloping rectangle associated with the . 
previous object box is also calculated. 

If the distance w is not greater than a predeter- 
mined critical value W. the enveloping rectangle O 
20 is extended so that it also contains the new object 
box, whereupon the following object box is consid- 
ered. If the distance w, on the other hand, is larger 
than the critical value W, then the enveloping rec- 
tangle associated with the previous object box," O, 
25 is closed and a new enveloping rectangle O' is 
formed around the new object box. This is shown 
in Fig. 5, where b'G) is the last object box of 
enveloping rectangle O and object box b'O + 1 ) is a 
horizontal distance w. larger than W. away from the 
30 enveloping rectangle O. The closed enveloping 
rectangle O is then handled as a subblock of block 
B. 

In this way the block B is divided into one or 
more adjoining subblocks. As^ a result of the»de-. 
35 scribed prior sorting the processing sequence of 
the object boxes is optimal so that the division into 
subblocks takes place very rapidly. 

After division of block B into subblocks by 
means of vertical cuts, each subblock is in turn 
40 divided into smaller blocks by means Qf horizontal 
cuts. This proceeds in a similar manner to that 
, described for vertical cuts, as follows: 

The procedure for the horizontal cutting of the 
subblock O from the abovedescribed division op- 
45 eration again starts with sorting the object boxes of 
the subblock, now* by ascending Y-coordinate of. 
their top left corner, and, for constant Y-coordinate, 
by ascending X-coordinate of their top left corner. 
The resulting series of object boxes thus starts with 
so the top box, and if different boxes have exactly the 
same Y-coordinate, the furthest left of those boxes. 

The smallest enveloping rectangle of the series 
is then determined for each following object box of 
the series, and the vertical distance h between the 
55 bottom of that object box and the enveloping rec- 
tangle associated with the previous object is also 
calculated. If the distance h is not larger than a 
predetermined critical value H, the enveloping rec- 
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tangle is extended to include the new object box as 
well, whereupon the following object box is consid- 
ered. . 

If, on the other hand, the distance h is larger 
than the critical value H t the enveloping rectangle 
associated with the previous object box is closed 
and a new enveloping rectangle is formed around 
the new object box. The closed enveloping rectan- 
gle is then handled as a subblock of subblock O. 

The choice of the critical values W and H 
governs the result of the cuts and should be adapt- 
ed to the level of the cut and to the size of the 
object boxes. For example, when a document page 
is divided into text columns, the value of W can be 
selected as being three times the average width of 
the object boxes, and for division of a line of text 
into words one and a half times the average dis- 
tance between the boxes in the line (this can be 
calculated per line during the procedure). These 
values are determined by experiment. 

The cuts in the X and Y directions also govern 
the corner points of the layout structure of a docu- 
ment page, such as paragraphs, text columns, 
lines, etc. The resulting layout structure can then 
be represented hierarchically in accordance with a 
tree structure. 

The method of determining whether the image 
data originates from a skew document and, if so, 
the magnitude of the skew angle, will now be 
discussed in detail with reference to Fig. 6. This 
method is used in step 20 in Fig. 3. 

The method determines the position of the 
lefthand margin and in so doing uses the object 
boxes of the "character" type in the object box file. 
This file contains all the object boxes with the 
coordinates of their top left and bottom right cor- 
ner. Where the following description refers to a 
box, it denotes a character object box. 

The method takes place in three phases: 

- searching for the box situated furthest left in 
each line 

- selecting from this set those boxes which 
realjy belong to the margin 

- calculating the skew angle from the direction 
of the margin. 

Once the skew angle has been determined, the 
object box file can be transformed so that the data 
therein relate to an aligned document. 

The first phase of the method starts with sor- 
ting the boxes by ascending X-coordinate and, for 
a constant X-coordinate ( by ascending Y-coordi- 
nate, of their top left corner. This is shown in Fig. 
6, where part of a skew text is shown rotated in the 
anticlockwise direction. Each character is provided 
with a box and a number of boxes is numbered 11 
to 17, in the sequence in which they are sorted. 

The boxes are then read out and included in 
the same sequence in a list provided they have no 



overlap with a box already included in the list 
previously or with the white space between that 
box and the box thereabove. This is effected by 
comparing the segment on the Y-axis occupied by 
5 the new box with the segments on the Y-axis which 
are always limited by the Y-coordinate of the bot- 
tom righthand corner of a given box already in- 
cluded, and that of the bottom righthand corner of 
the already included box situated directly 
io thereabove. Since the second and following char- 
acters of a line always overlap the first character, 
even in the event of a skew position, and can thus 
be eliminated, on completion the list will only con- 
tain the first box of each line. The first elements of 
is the list are now thus formed by the boxes 11, 12, 13 ( 
14, 15, 17, etc., in Fig. 6, since box 16 overlaps box 11 
and is thus not included in the list. 

The boxes in the list are then resorted, now by 
ascending value of the Y-coordinate of their top 
2 o lefthand corner. They are then read out from the 
list one by one and the direction vector from their 
top left corner to the top left corner of the next box 
from the list is calculated from the differences dx, 
dy of their X and Y coordinates. -This is shown in 
25 Fig. 6, where the arrows always represent the di- 
rection vectors. This direction vector is. now com- 
pared with that of the preceding box, and if its 
direction does not deviate therefrom by more than 
a value V, there is a chance that a margin has 
30 been found. There can incidentally be more than 
one margin on a page. e.g. in the case of an 
indented paragraph. Starting from a predetermined 
maximum permissible deviation in the X-direction 
between two characters situated at corresponding 
35 positions in two consecutive text lines, the value V 
is recalculated for each box as a function of the 
difference in Y-coordinate between the two boxes 
between which the direction vector is calculated. 
,. The idea underlying this is as follows. Even 

40 when a text is completely aligned, there may be : 
small differences in X-position between two char- 
acters situated one under the other. These differ-, 
ences are increased by a skew position. This is the 
result of differences' in form. The maximum de- 
45 viation is made the same for all lines, e.g. 3 image 
dots. On the basis of a line spacing, for example, 
of 35 image dots the maximum deviation is then 
4.9 for two consecutive lines, but only 1.6 over 
three lines. The value of V will thus be smaller in 
so proportion as the two boxes between which the 
direction vector is calculated are farther away from 
one another in the Y-direction. 

After the analysis of the boxes in the list, a 
search is then made for the longest row of boxes 
55 whose direction vectors are equal within the toler- 
ance V and this is then considered as (part of) the 
margin. From the two end points of the row a 
search is then made for other boxes situated out- 
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side the row and also situated on the margin. This 
would apply, for example, if there is an indented 
paragraph in the middle of a page. There are then 
two page parts, above and below the indented 
paragraph, situated on the same margin. The direc- 
tion vector is first determined from the bottommost 
box of the row to the box beneath it and a check is 
made whether it is equal, within the tolerance V. to 
that applicable to the margin boxes. If that is not so 
(and it will certainly be so for the first box exam- 
ined), then the box beneath it is tested, and so on, 
till a box is found whose direction vector does 
come within the tolerance V. This box is again 
situated on the margin and is therefore included in 
the row of margin boxes. Similarly, a search is 
made in the other direction from the row of margin 
boxes for other margin boxes. 

When all the margin boxes have been found, a 
straight line is drawn through their top left corners 
(e.g. by means of the smallest square method) and 
the skew is calculated from the direction of the 
straight line. In practice, it is possible with this 
method to determine a. skew accurately to approxi- 
mately 0.2 . 

The processing of the image data in memory 
208 (Fig. 2). in connection with the layout analysis, 
will now be discussed with reference to Fig. 7, 
which shows the control panel 1 05. 

After the digital image information of a docu- 
ment (page) has been stored in the internal mem- 
. ory 208 of the apparatus, it is automatically seg- 
mented by segmentation module 209. The result- 
ing layout structure is also stored in memory 210. 
This structure contains the locations of all the char- 
acter object boxes, and of all the higher elements 
formed by the text object boxes, such as blocks, 
lines and words, in addition to the locations and 
type of all the other object boxes. 

An image of the document, on which the object 
boxes are also drawn, is then shown, reduced if 
required, on the display of the display unit 102. 
Using the light pen 103. the operator can now 
indicate elements on the display. He can indicate 
which element he wishes to select by one of the 
keys on the control panel 105 (Fig. 5) under the 
heading SELECTION LEVEL. If the , operator 
presses the WORD key and indicates a (text) ob- 
ject box with the light pen, then memory 210 
searches for the element of the "word" type, of 
which the indicated object box forms part. This 
element then appears on the display in framed 
form or otherwise conspicuously. Similarly, the 
keys BLOCK and LINE refer to elements of the 
"block" and "line" type respectively while OBJECT 
and PAGE refer to a single object box and the 
whole page respectively. Finally, the AREA key can 
be used to select a rectangular frame whose di- 
mensions follow the movement of the light pen 



(known as "rubber banding" in personal computer 
technology). 

Once a layout element has been selected, it 
can be processed by processing module 212, from 

5 the series indicated under the heading OPERA- 
TION on the right of the control panel. 

If SET COLOUR is selected, the selected ele- 
ment is printed in a different colour. The UP and 
DOWN keys can be used to make a choice from a' 

ro number of available colours. Information as to the 
possible choices then appears on the display. The 
processing module 212 then places a code in the 
memory 208 at the image dot data of the selected 
layout element, to instruct the printing unit 207 to 

T5 print the relevant part of the image in the required 
colour. 

The SET CONTRAST operation can be used to 
control the reproduction characteristic (gradation) of 
the print for the selected element, the UP and 
20 DOWN keys making the reproduction harder and 
softer respectively: This is important, for example, 
if the original image is not sufficiently contrasty to 
give a clear image. In that case the processing 
module 212 selects a different conversion function 
25 from grey value image data to binary (white/black) 
image data for printing purposes during the hal- 
ftoning operation. ■ , 

The selected document can be enlarged or 
reduced with the RESIZE operation (using the UP 
30 and DOWN keys), while INVERT is used to print it 
in negative form and ROTATE is used to rotate it in 
the image (the angle of rotation can be selected 
with the UP and DOWN keys). 

The CUT & PASTE heading contains keys for 
35 carrying out operations at the location of a selected 
document. In the case, of DUPLICATE, a. selected 
element is copied at the place indicated by the 
light pen. With MOVE a selected element can be 
moved ("dragged") across the display while a se- 
40 lected element can be removed from -the image 
with REMOVE. . ^ 

Finally, the heading INCLUDE contains the 
keys LOAD for loading image data from an external 
source, and CREATE PAGE for compiling a docu- 
45 ment page, for which it is possible to use the keys 
from the CUT & PASTE heading. In the case of this 
last function, an empty page image is shown on 
the display next to the page already selected and 
layout elements from the selected page can be 
so - transferred to the empty page in order thus to 
compile a new image, e.g. a clippings newssheet. 

The abovedescribed functions are performed 
by a computer accommodated in the reproduction 
apparatus. The programming required for the pro- 
55 cessing of layout elements is quite well known from 
computer and PC technology and is therefore not 
explained in detail here. 
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The invention has been explained with refer- 
ence to the above description of an embodiment, 
but it will be clear to the skilled man that other 
embodiments also come within the scope of the 
following claims. 

Claims 

1. Apparatus for reproducing an image, compris- 
ing operating means, means for supplying digi- 
tal image dot information representing said im- 
age , a memory for storing image dot informa-. 
tion, means for processing the supplied image 
dot information and means for printing the pro- 
cessed image dot information on an image 
support, 

. characterised by 

segmentation means, connected to the mem- 
ory, for segmenting the digital image dot in- 
formation into layout elements, 
the operating means being connected to the 
segmentation means for receiving therefrom 
results of the segmentation and being provided 
with: 

- means for displaying said results to an 
operator, 

- first selection means for selecting one or 
more layout elements found during the 
segmentation, 

- second selection means for selecting a 
processing operation for changing the 
appearance of said image, 

said operating means also being connected to 
the processing means for controlling the same 
in order to process a selected layout element 
in accordance with a selected processing op- 
eration. 

. 2. Apparatus according to claim 1, wherein the 
- processing .operation comprises changing the 
grey value scale. 

3. Apparatus according to claim 1, wherein the 
processing .operation comprises changing the 
colour in the print on the image support. 

4. Apparatus according to claim 1, wherein the 
' processing operation comprises the introduc- 
tion of a raster. 

5. Apparatus according to claim 1, wherein the 
processing operation comprises changing the 
orientation. 

6. Apparatus according to claim 1, wherein the 
processing operation comprises changing the 
location. 



7. Apparatus according to claim 1, wherein the 
segmentation means are adapted to search the 
digital image dot information for objects 
formed by clusters of adjoining information- 

5 bearing image dots, and perform the segment- 

ing into layout elements on the basis of said 
objects. 

8. Apparatus according to claim 7, wherein the 
w segmentation means assign to each object a 

frame, hereinafter referred to as an object box. 
having two horizontal and two vertical sides 
and, for each object, store the location data of 
the object box in a memory. 

75 9. Apparatus according to claim 7, wherein the 
segmentation means assign an object label to 
an object, said object label being representa- 
tive of the type of information of said object. 

20 ... 
'10. Apparatus according to claim 9. wherein the 
first selections means are adapted to select all 
objects having a specific object label at once. 



25 11. Apparatus according to claim 9, wherein the 
segmentation means only make use of objects 
having predetermined object labels in the said 
segmenting. 

30 12. Apparatus according to claim 11, wherein the 
said predetermined object labels are those la- 
bels which indicate that the object represents a 
character. 

35 13. Apparatus according to any one of claims 7 to 
12, wherein prior to said segmenting the seg- 
. mentation means first determine whether' and, ; 
if so, to what extent the page image is skew 
with respect to the vertical and horizontal di- 

40 rections, and if so align the set.of objects by 

adapting the location data of the objects in 
accordance with a rotation through the skew 
angle. 

45 14. A method of segmenting into layout elements 
a page image in digital image dot information 
form, comprising dividing the page image into 
eversmaller blocks in consecutive operations 
by means of cutting operations in a first and 

so second predetermined direction alternately, 

characterised in that 

objects each formed by. a cluster of adjoining 
informationbearing image dots are first 
searched for in the digital image dot informa- 
55 tion and 

in that the objects are considered as basic 
elements in the said cutting operations. 



9 



17 



EP 0 629 078 A1 



15. A method according to claim 14, characterised 
in that each object is assigned a frame, 
hereinafter referred to as an object box. having 
sides parallel to the said directions, said frame 
just enclosing the object, and that for determin- 
ing the location of an object within the page 
image information is taken from the location of 
the object box of said object. 

16. A method according to claim 14 or 15, charac- 
terised in that an object is assigned an object 
label which is a representative of the type of 
information of that object. 

17. A method according to claim 16, wherein only 
objects having predetermined object labels are 
used for the said cutting operations. 

•18, A method according to claim 17, wherein the 
said predetermined object labels are those la- 
bels which indicate that the object represents a 
character. 

19. A method according to any one of claims 14 to 

18, characterised in that a. cutting operation 
divides a block into subblocks with the criterion 
that the objects of a first subbiock are further 
away from an adjoining subbiock than a pre- 
determined distance. 

20. A method according to any one of claims 14 to 

19, characterised in that prior to the cutting 
operations it is first determined whether. and, if 
so, to what extent the page image is skew with 
respect to a cutting direction, and in that if so 
the page image is first aligned by displacing 
the objects in accordance with a rotation 
through the skew angle. 
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